Skip to content

fix(testing): reduce flakiness of terminal.test.ts and use 1 worker for e2e tests #3263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 30, 2021

Conversation

jsjoeio
Copy link
Contributor

@jsjoeio jsjoeio commented Apr 29, 2021

This PR reduces the flakiness of the terminal.test.ts and testing overall by using 1 worker. It doesn't eliminate all flakiness but should reduce it.

Changes

Screenshots

Running the terminal.test.ts 30 times locally (10 for each major browser) and it only failed once -> in chromium.
image

It's failing because it's either not receiving the full echo command or hitting enter before it receives the full command.

7ea99daf58ae2eb51a8c2a83742b7b9f.mp4

Checklist

  • tested locally

I'm going to open up a separate PR to add an additional check for e2e tests to check the connection. #3270

@jsjoeio jsjoeio self-assigned this Apr 29, 2021
@jsjoeio jsjoeio added the testing Anything related to testing label Apr 29, 2021
@jsjoeio jsjoeio added this to the v3.9.4 milestone Apr 29, 2021
@jsjoeio jsjoeio changed the title jsjoeio/fix-e2e-test fix(testing): reduce flakiness of terminal.test.ts and use 1 worker for e2e tests Apr 29, 2021
@codecov
Copy link

codecov bot commented Apr 29, 2021

Codecov Report

Merging #3263 (7309ea9) into main (0d9fe6f) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #3263   +/-   ##
=======================================
  Coverage   46.90%   46.90%           
=======================================
  Files          23       23           
  Lines        1196     1196           
  Branches      237      237           
=======================================
  Hits          561      561           
  Misses        451      451           
  Partials      184      184           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d9fe6f...7309ea9. Read the comment docs.

@jsjoeio jsjoeio marked this pull request as ready for review April 30, 2021 19:02
@jsjoeio jsjoeio requested a review from a team as a code owner April 30, 2021 19:02
Copy link
Contributor

@jawnsy jawnsy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sweet!

@@ -52,6 +52,7 @@ const config: Config = {
testDir: path.join(__dirname, "e2e"), // Search for tests in this directory.
timeout: 60000, // Each test is given 60 seconds.
retries: 3, // Retry failing tests 2 times
workers: 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this make tests slower?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this make tests slower?

Yes 😢 Unfortunately, it seems having these tests run in parallel against the same code-server application leads to flakiness (i.e. weird things happen). It's probably the way code-server is built and/or having concurrent connections to the same instance.

More about workers:

Each worker process creates a new environment to run tests. Different environments always run in different workers. By default, Folio reuses the worker as much as it can to make testing faster, but it will create a new worker when retrying tests, after any test failure, to initialize a new environment, or just to speed up test execution if the worker limit is not reached.
folio docs (playwright-test uses folio under the hood)

@jsjoeio jsjoeio merged commit 8e6c10e into main Apr 30, 2021
@jsjoeio jsjoeio deleted the jsjoeio/fix-e2e-test branch April 30, 2021 22:38
jsjoeio added a commit that referenced this pull request Oct 7, 2021
I don't know if it's a resources issue, playwright, or code-server but it seems
like the e2e tests choke when multiple workers are used.

This change is okay because our CI runner only has 2 cores so it would only use
1 worker anyway, but by specifying it in our playwright config, we ensure more
stability in our e2e tests working correctly.

See these PRs:
- #3263
- #4310
code-asher added a commit that referenced this pull request Oct 28, 2021
* fix(testing): revert change & fix playwright tests

* fix(constants): add type to import statement

* refactor(e2e): delete browser test

This test was originally added to ensure playwright was working.

At this point, we know it works so removing this test because it doesn't help
with anything specific to code-server and only adds unnecessary code to the
codebase plus increases the e2e test job duration.

* chore(e2e): use 1 worker for e2e test

I don't know if it's a resources issue, playwright, or code-server but it seems
like the e2e tests choke when multiple workers are used.

This change is okay because our CI runner only has 2 cores so it would only use
1 worker anyway, but by specifying it in our playwright config, we ensure more
stability in our e2e tests working correctly.

See these PRs:
- #3263
- #4310

* revert(vscode): add missing route with redirect

* chore(vscode): update to latest fork

* Touch up compilation step.

* Bump vendor.

* Fix VS Code minification step

* Move ClientConfiguration to common

Common code must not import Node code as it is imported by the browser.

* Ensure lib directory exists before curling

cURL errors now because VS Code was moved and the directory does not
exist.

* Update incorrect e2e test help output

Revert workers change as well; this can be overridden when desired.

* Add back extension compilation step

* Include missing resources in release

This includes a favicon, for example.  I opted to include the entire
directory to make sure we do not miss anything.  Some of the other
stuff looks potentially useful (like completions).

* Set quality property in product configuration

When httpWebWorkerExtensionHostIframe.html is fetched it uses the web
endpoint template (in which we do not include the commit) but if the
quality is not set it prepends the commit to the web endpoint instead.
The new static endpoint does not use/handle commits so this 404s.

Long-term we might want to make the new static endpoint use commits like
the old one but we will also need to update the various other static
URLs to include the commit.

For now I just fixed this by adding the quality since:
  1. Probably faster than trying to find and update all static uses.
  2. VS Code probably expects it anyway.
  3. Gives us better control over the endpoint.

* Update VS Code

This fixes several build issues.

* Bump vscode.

* Bump.

* Bump.

* Use CLI directly.

* Update tests to reflect new upstream behavior.

* Move unit tests to after the build

Our code has new dependencies on VS Code that are pulled in when the
unit tests run.  Because of this we need to build VS Code before running
the unit tests (as it only pulls built code).

* Upgrade proxy-agent dependencies

This resolves a security report with one of its dependencies (vm2).

* Symlink VS Code output directory before unit tests

This is necessary now that we import from the out directory.

* Fix issues surrounding persistent processes between tests.

* Update VS Code cache directories

These were renamed so the cached paths need to be updated.  I changed
the key as well to force a rebuild.

* Move test symlink to script

This way it works for local testing as well.

I had to use out-build instead of out-vscode-server-min because Jest
throws some obscure error about a handlebars haste map.

* Fix listening on a socket

* Update VS Code

It contains fixes for missing files in the build.

* Standardize disposals

* Dispose HTTP server

Shares code with the test HTTP server.  For now it is a function but
maybe we should make it a class that is extended by tests.

* Dispose app on exit

* Fix logging link errors

Unfortunately the logger currently chokes when provided with error
objects.

Also for some reason the bracketed text was not displaying...

* Update regex used by e2e to extract address

The address was recently changed to use URL which seems to add a
trailing slash when using toString, causing the regex match to fail.

* Log browser console in e2e tests

* Add base back to login page

This is used to set cookies when using a base path.

* Remove login page test

The file this was testing no longer exists.

* Use path.posix for static base

Since this is a web path and not platform-dependent.

* Add test for invalid password

Co-authored-by: Teffen Ellis <[email protected]>
Co-authored-by: Asher <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Anything related to testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants